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Abstract —This paper addresses the problem of matching N weighted 
graphs referring to an identical object or category. More specifically, 
matching the common node correspondences among graphs. This 
multi-graph matching problem involves two ingredients affecting the 
overall accuracy: i) the local pairwise matching affinity score among 
graphs; ii) the global matching consistency that measures the unique¬ 
ness of the pairwise matching results by different chaining orders. Previ¬ 
ous studies typically either enforce the matching consistency constraints 
in the beginning of iterative optimization, which may propagate matching 
error both over iterations and across graph pairs; or separate affinity 
optimizing and consistency regularization in two steps. This paper is 
motivated by the observation that matching consistency can serve 
as a regularizer in the affinity objective function when the function is 
biased due to noises or inappropriate modeling. We propose multi¬ 
graph matching methods to incorporate the two aspects by boosting 
the affinity score, meanwhile gradually infusing the consistency as a 
regularizer. Furthermore, we propose a node-wise consistency/affinity- 
driven mechanism to elicit the common inlier nodes out of the irrelevant 
outliers. Extensive results on both synthetic and public image datasets 
demonstrate the competency of the proposed algorithms. 

1 Introduction 

RAPH matching (GM) ||5|, |hl has received exten¬ 
sive attentions over decades, and has found wide 

A preliminary version of the presented paper appeared in [ij. The current 
paper makes several extensions and improvements: i) propose a new graduated 
consistency-regularized affinity score boosting algorithm (ISB-GC in 
which in general achieves more accurate matching results, meanwhile helps 
reinterpret the method (ISB-GC-U in of the conference paper; ii) 

improve the robustness of the methods in face^ outliers by integrating a node¬ 
wise consistency/affinity-driven inlier eliciting mechanism; Hi) present more 
technical details of the proposed algorithms and convergence discussion which 
are not fully described in 0; iv) perform extensive evaluations that involve 
more various settings and additional datasets. In addition, more emerging 
state-of-the-arts j2j, 0, 0 are compared especially after the year 2013. The 
source code of our methods will be made public available. 
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applications in various problems such as bioinformat¬ 
ics EI, data fusion |I8|, graphics |9| and information 
retrieval mni, name a few. In particular, GM lies at 
the heart of a range computer vision tasks as diverse 
as object recognition, shape matching, object tracking, 
and image labeling among others, which require finding 
visual correspondences - refer to [4j for more specific 
references. Different from the point based matching or 
registration methods such as RANSAC ITU and Iterative 
Closet Point (ICP) |[T2|, GM incorporates both the unary 
node-to-node, and the second-order edge-to-edge structural 
similarity. By encoding the geometrical information in 
the representation and matching process, GM methods 
are in general supposed to be more robust for solving the 
correspondence problems. Due to its well-known NP- 
complete nature, existing GM methods involve either 
finding approximate solutions 1131 , 1141 , ITSl or obtaining 
the global optima in polynomial time for a few types 
of graphs, including the planar graph ||T6|, bounded 
valence graph |T3 and tree structure iTSl . 

Most GM methods focus on establishing one-to-one 
correspondences (which is also the interest of this pa¬ 
per) between a pair of feature points 113, [201, EH/ 
ED, GZI- In general, the pairwise GM problem can be 
formulated as a quadratic assignment problem (QAP) 
that accounts for both individual node matches (unary 
terms) and pairs of matches (pairwise terms). The QAP 
formulations can be roughly divided into two sub¬ 
categories E3' i) the Koopmans-Beckmann's QAP l23l 
tr(X^AiXA 2 )+tr(KjX) where X refers to the assignment 
matrix between two graphs as will be introduced later in 
the paper. Ai, A 2 are the weighted adjacency matrices 
and Kp is the node-to-node similarity matrix between 
two graphs; ii) the more general Lawler's QAP problem 
1^ by vec(X)^Kvec(X) where K is the affinity matrix 
encoding unary and second-order edge affinities. The 
former can always be represented as a special case of 
the Lawler's QAP by setting K = A 2 (8) Ai. 

Nevertheless, in many applications, visual objects do 
not appear in isolation or in pair, but more frequently in 
collections or families. Such collections provide a context 
and potentially allow for higher quality matching and 
analysis. Given a batch of graphs referring to an identical 
or related structure, it is required to find the global 
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matchings across all graphs. Due to the advance of 
modern imaging and scanning technologies, multi-graph 
matching is applied to infusing multi-source sensor data 
|8l|. Graphic shape analysis and search often require to 
model objects by multi-view assembly 1251 . And ITOl 
applies multi-graph matching to the problem of multi¬ 
source topic alignment. It is also related to graph clus¬ 
tering, classification and indexing. Several multi-graph 

matching methods E3, EZI/ ESI/ EH/ ESI/ Gl/ El are 
proposed, which will be detailed in the next section. 

2 Related Work and Models 

Pairwise graph matching The problem of pairwise GM 
involving two graphs has been extensively studied in 
the literature. It is beyond the scope of this paper for an 
exhaustive summary of these work. As will be detailed 
later in this paper, our methods and many other multi¬ 
graph matching methods EH/ IH/ [21/ ESI build on top of 
pairwise matching solvers as a black-box, regardless the 
specific context in which these pairwise GM techniques 
are devised, such as machine learning based pairwise 
GM approaches GH, ED/ ED/ E2/ ESI/ hyper-graph 
matching approaches (ID/ (SD/ (S3/ (SD/ (^ and the 
recent work |3l tailored for massive outliers. 

Most pairwise GM methods t38|, ||39|, (l0|, ilSI , 
Gl/ (23 are based on the Lawler's QAP. Given two 
graphs Qi and Q 2 of node size ni and 77 . 2 , an affinity 
matrix K G xnms jg (Jefined such that its ele¬ 
ments 6=1 uTeasure the edge pair affinity 

{{vi,Vj) ^ two graphs. The diag¬ 
onal term 1^2 1 ^ 0=1 describes the unary affinity of 

a node match {vi ^ 1 ^ 0 = 1 - More rigorously, most 

existing graph matching work [H, GH, (131, Gl follow 
a convention for setting K such that the element Kia-jb 
for the edge pair {vi,Vj) ^ (va.Vb) is located at the ((a- 
l)ni+i)-th row and (( 6 -l)n 2 +j)-th column of K. 

We define the assignment matrix X G 
for two graphs which establishes the one-to-one node 
correspondence such that Xia = 1 if node Vi matches 
node Va (0 otherwise). The problem of GM involves 
finding the optimal correspondence X such that the sum 
of the node and edge compatibility between two graphs 
is maximized. Without loss of generality, similar to GH/ 
(131, by assuming ni > n 2 for different sizes of graphs, it 
leads to the following widely used two-way constrained 
quadratic assignment problem (QAP) {e.g. GD/ Gll and 
the references therein): 

X* = argmaxvec(X)^Kvec(X) (1) 

S.t. Xln^Kln, = Xe 

The constraints refer to the two-way one-to-one node 
mapping: a node from graph Qi can match at most one 
node in Q 2 and every node in Q 2 is corresponding to one 
node in Qi. There is no (one/many)-to-many matching. 

The above formulation is as general as it allows two 
graphs having unequal number of nodes (ni 7 ^ 722 ). 
A common protocol adopted by the previous studies 
EBl, (T 3 I , (41 is converting X from an assignment matrix 
(^ln 2 ^ Ini) to a permutation matrix (Xln^ = InJ 


by adding dummy nodes to one graph (i.e. adding 
slack variables to the assignment matrix and augment 
the affinity matrix by zeros) in case ni 7 ^ 77 , 2 . This is 
a common technique from linear programming and is 
widely adopted by (33, GH such that is supposed can 
handle superfluous nodes in a statistically robust manner 
(see the last paragraph of Sec.2.3 in (38l ). By taking this 
step, the graphs are of equal sizes. This preprocessing 
also opens up the applicability of existing multi-graph 
methods (301 , (28l . as they all assume that all graphs 
are of equal sizes. Therefore, the following formulation 
is derived which assumes 77.1 = 77.2 = n. 

XI 2 = argmaxvec(Xi2)^Ki2vec(Xi2) (2) 

X12 

s.t. Xi2ln2 = Ini luiXl2 = G, X12 € { 0 , 

X 12 is a permutation matrix constructed by augmenting 
the raw matching matrix with slack columns if tt-i > 77 , 2 . 
This is also coherent with the ideal case for the problem 
setting of this paper, matching the common inliers of 
equal sizes among graphs in the absence of outliers, or 
they are dismissed by a certain means. 

We also mention several other threads for pairwise 
GM: i) learning the affinity over graphs II 2 TI , (^, (191 , 
E2, E3|; ii) incorporating higher-order information (3D , 
(Ml , (35l , [36J, |3Z|; hi) progressive matching f^| or com¬ 
bining feature detection and matching in an integrated 
system ESI. These methods focus on pairwise matching 
and are methodologically less relevant to this paper. 

Multi-graph matching Recently, matching a collection 
of related graphs becomes an emerging research topic 
with researchers from different communities including 
computer vision (23 , fD/ machine learning (23, [2|, 
pattern recognition El, [261, [301/ graphics [91 and in¬ 
formation retrieval cqi, among others. We divide the 
state-of-the-arts into two categories concerning how the 
affinity and matching consistency are explored. 

i) affinity score-driven approaches (33/ EH/ (D/ O' 
For these methods, usually first a compact set of basis 
pairwise matching variables are generated which can 
derive the matching solutions for each pair. Then an 
objective function regarding the overall pairwise match¬ 
ing affinity score is maximized by different algorithms. 
Sole-Ribalta et ah (33 extend the algorithm Graduated 
Assignment (GA) (381 for two-graph case to multi-graph. 
Each graph is associated with an assignment matrix 
mapping the nodes to a virtual node set. Then the vari¬ 
able set is updated in a deterministic annealing manner 
to maximize the overall affinity score. Yan et al. ESI/ SI 
adaptively find a reference graph which induces a 
compact basis matching variable set {Xkr}k=i^^r 

N graphs. An alternating optimization framework is 
devised to update these basis variables in a rotating 
manner. The early work (Ml by Gavril et al selects a 
set of basis variables from the initial pairwise matching 
solutions via finding the maximum spanning tree on the 
super graph related to a maximized overall affinity score. 
In this sense, their work also falls into this category. 

ii) pairwise matching consistency-driven approaches (4D , 
(23/ E3/ (21 • these methods usually are comprised of 
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two steps. First, a pairwise GM solver is employed to ob¬ 
tain the node mapping between all (or a portion of) pairs 
of graphs. Then, a spectrum smoothing technique is de¬ 
vised to enforce global matching consistency in the sense 
that two sequential pairwise matching in different chain¬ 
ing orders shall lead to the same solution. Specifically, in 
mi and the journal version ||26|, Sole-Ribalta et al. use 
a hyper-cube tensor to represent the A^-node matching 
likelihood, each from N different graphs. Then a greedy 
method is used to binarize the final solutions satisfying 
the one-to-one two way constraints. Pachauri et al. (261 
employ spectral analysis via eigenvector decomposition 
on the input pairwise matching solutions, and recover 
the consistent matching solutions. The success of this 
method lies on the assumption that the input pairwise 
matchings is corrupted by Gaussian-Wigner noise which 
is too ideal in reality. A more recent work proposed by 
Chen et al. [2| is suitable when only a part of nodes can 
find their correspondences in other graphs, which they 
term as 'partial similarity'. They formulate the problem 
into a tractable convex programming problem solved by 
the popular first-order Alternating Direction Method of 
Multipliers (ADMM) method tailored for their semidef- 
inite programming problem. 

One observation to the above mentioned approaches 
is that they either enforce consistency early by using a 
compact set of basis variables for pairwise matchings 
m, a m, m, which runs at the risk of propagating 
errors across iterations and graphs, or assume the initial 
pairwise matchings are obtained with corruptions by 
another method, and focus on improving the overall 
accuracy from the input pairwise matchings via spectral 
methods, however no affinity information is used in the 
procedure EH, |26l, pS), [2|. 

In particular, a^wi^ly used multi-graph matching 
formulation |29|, |41, [30J is as follows, under the implicit 
assumption of invertible pairwise matching relations i.e. 
Xij = Xj-, which meanwhile enforces matching consis¬ 
tency over all pairwise matchings: 

N-l N 

vec(Xi^)^Ki^vec(Xi^) (3) 

i=i j=i+i 

S.t. = ln,,Xzi = G {0, 

Note the above formulation assumes each graph shall 
contain and only contain the common inlier nodes, 
which does not hold in the presence of unmatchable 
outliers. Thus it calls for more flexible solutions. 

In this paper, we adopt a flexible approximate affinity 
score boosting procedure, which is gradually regularized 
by the overall matching consistency. The key idea is 
that the affinity score is indicative to semantic matching 
accuracy in the early stage of iterative boosting, while 
it becomes less informative as the accuracy lifting satu¬ 
rates. Then, in the presence of a few outliers, consistency 
becomes a useful regularizer to avoid the degenerating 
case between the two local graphs due to the fact that 
affinity score is often biased to semantic accuracy when 
the two graphs are corrupted by arbitrary noises, in 


addition with the inherent difficulty in modeling the 
affinity matrix using a compact parametric model. In 
the presence of more outliers, we propose a node¬ 
wise consistency/affinity-driven mechanism to elicit the 
affinity score and consistency relevant to common inlier 
nodes. Extensive empirical results illustrate the efficacy 
of our methods which is further specified in our conclu¬ 
sive remarks in SeclU 

3 Proposed algorithms 

Given the initial pairwise matching configuration X(o) 
generated by a pairwise matching solver e.g. 1141 , ITSl , 
one can further derive a new pairwise matching by 
the composition of a few 'good' matchings i.e. X^y = 

to replace the original such 
that the matching accuracy can be improved or ide¬ 
ally maximized, by means of interpolating the sequence 
ft, 0/ci, • • • , Gks-i ^ Gks ^ Gj- The problem of this dynamic¬ 
replacing framework is how to set up the appropriate 
compositional replacements. In case the original match¬ 
ing is best, the replacement set shall be empty. 

This paper devises a mechanism for iteratively finding 
the appropriate new composition to replace the old 
one without knowing their true accuracy. We will start 
with a baseline ISB (Algj^, and two variants 
ISB^st fQj, comparison. Then a graduated consistency- 
regularized method ISB-GC (Alg|^ is highlighted with 
two efficient variants ISB-GC-U/P (Alg|^. Before diving 
into the details, we first introduce several notations and 
definitions which help the exposition of our methods. 

3.1 Notations and definitions 

Throughout the paper, M denotes for the real number 
domain. Bold capital letters denote for a matrix X, bold 
lower-case letters for a column vector x, and hollow 
bold letters for a set X. All non-bold letters represent 
scalars. X^ is the transpose of X; vec(X) is the column¬ 
wise vectorized matrix X. tr(X) is the trace of matrix X. 
In G is an identity matrix and the subscript n 

will be omitted when it can be inferred from context. 
Imxn, Omxn ^ is the matrix whose elements being 

all ones or zeros, respectively. In is the abbreviation for 
Inxi- |X| is the cardinality of the set X, ||X||i7 = tr(X^X) 
for the Frobenious norm and || • \\p is the p-norm. 

This paper intensively uses X = to 

denote the set of pairwise matching matrices over N 
graphs {Gk}k=i^ which is also termed as matching con¬ 
figuration, and K = {^ij}i^j=i is used for the affinity 
matrix set. In particular, we use Xc to denote a solution 
set which is fully consistent (referred by the subscript 
'c') as there is no contradictory pairwise matchings for 
{Xij ^ ^ik^kj}i^j^k=i' otherwise stated explicitly, 

we adopt the convention that uses N for the number of 
considered graphs, n and m for the number of nodes, 
and the number of edges in a graph, respectively. 
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So far we have not presented a formal definition 
regarding consistency, which has been widely used in 
E9l, 111, El, 1261 , 1281 and mentioned earlier in this paper. 
Now we give the formal definitions regarding matching 
consistency induced by the initial pairwise matching X 
from any pairwise matching solver. In addition, we also 
define two variants of the super graph and a concept 
related to matching composition. This paper does not 
claim the credit for the novelty of these definitions as 
similar definitions or concepts may appear in literature. 

Definition 1. Given N graphs {Gk}^i the pairwise 
matching configuration X = the unary 

consistency of graph Qk is defined as Cu{k,X) = 1 — 
11^/2 ^ 

nN{N-l)/2 

Definition 2. Given graphs {Gk}k=i ^^d matching configu¬ 
ration X, for any pair Gi and G^, the pairwise consistency is 

defined as Cp{Xij,X) = 1 - ^ 

Definition 3. Given N graphs {Gk}J^i and X, we call X 
is fully consistent if and only if ^ ^^d 

(or) ^k=i^u(k,x) ^ The following equation always holds: 
ELig~(fe’X) ^ ^ /Mrf/jer define the 

overall consistency o/X as C(X) = e (0,1]. 

Definition 4. Given {Gk}k=i ^^d X,for node in 

graph Gk, ds consistency w.r.t. X is defined by Cn{u^,X) = 

^ ^ ^ -XkiXij 

and Y(u ^,:) is the u^th row of matrix Y. 

Definition 5. Given {Gk}k=v IK,/or node in 

Gk, ds affinity w.r.t. X and K is defined by X,K) = 

^k )^^kinec{Xki)r where X^ denotes the ma¬ 

trix X with zeros except for the u^-th rows as is. 

Definition 6. Given {Gk}k=i ^^d X, the affinity-wise super 
graph Gsup d defined as an undirected weighted graph s.t. 
each node k denotes Gk, and its edge weight Cij is the ^finity 
score Jij{Xij) = vec{Xij)^KijVec{Xij) induced by ^ 

Definition 7. Given {Gk}k=i ^^d X, the consistency-wise 
super graph Gsup d defined as an undirected weighted graph 
s.t. each node k denotes Gk, and the weight of edge Cij is the 
pairwise consistency Cp{Xij,X) = 1 - ^fc=i W^ij-^ikXkjWr/^ ^ 

Definition 8. The path Zij{ki, k 2 ,... ,ks) from Gi to Gj is 

defined as the chain Gi^Gki^ - >Gks^Gj 'o^hich induces: 

^2 5 • • • 5 ^s) — Xik-fXk-^k 2 • • ’^ksj’ order Sij is the 
number of the intermediate graphs between Gi, Gj. The path 
score is Jij{ki, fe,..., ks) =vec{Yij)^KijVec{Yij). 

We present several comments to the above definitions. 
The unary consistency Cu{k,X) by Definition Q is a 
function w.r.t. the graph index k and X. In contrast, 

1. In general Jij{Xij) / Jji{Xji) and |45l, |30l also add Jij(Xij) 
into the objective. While this setting convention is not related to the 
essence of our methods, thus for efficiency we use one-way affinity 
score to set the affinity-wise super graph and the objective in Eq|^ 


the pairwise consistency Cp{Xij,X) by Definition (|^ is 
generalized to the one w.r.t. two variables of Xij and X 
and the former one Xij does not necessarily belong to 
X. The pairwise consistency metric is symmetric such 
that Cp{Xij,X) = CpiXji.X), which derives Definition 
Definition and <0 break down to node level 
consistency/affinity similar to unary ones, and they will 
be used in the proposed inlier eliciting mechanism. 

The super graph, either for affinity-wise Gsup by Def¬ 
inition 0, or consistency-wise Ggup by Definition 0, is 
a connected (not necessarily fully connected) gra^ as 
a portion of pairs are matched by a certain means, a 
maximum spanning tree (MST) HU can be found with no 
less total weight of every other spanning tree. 

Note Definition 0 and Definition 0 appeared in [H 
in which they are used to identify the optimal alternating 
optimization sequence of variables. This paper utilize 
them differently to infuse the consistency along with the 
boosting procedure. The overall consistency for X as by 
Definition 0, further tells the relation between unary 
consistency and pairwise consistency. A similar overall 
'inconsistency' metric appears in [451, [30J. 

The affinity-wise super graph Gsup by Definition 0 is 
similar to the one in i46l , of which the author proposes 
to build a graph where each shape is a vertex, and edges 
between shapes are weighted by the cost of the best 
matching. We put it in the scenario for the problem of 
weighted multi-graph matching. Furthermore, we give a 
similar definition for a consistency variant Ggup terms 
of consistency-wise edge weights by Definition 0. 

Finally, Definition 0 is related to the approximate 
path selection idea as used in our algorithms. Obviously, 
when X is fully consistent, any two paths between two 
given graphs would induce the equal solution. 

Based on the above definitions and discussion, we 
present our main approaches in the rest of this section. 

3.2 Iterative affinity score/consistency boosting 

Most GM methods aim to maximize an objective func¬ 
tion regarding with affinity score 12^ , [41, IT^ , 1131 , ITSl , 
we also follow this methodology in this subsection. Our 
basic rationale is that the node matching regarding with 
the highest affinity score between two graphs can be 
found along a higher-order path, as related to Definition 
0, instead of the direct (zero-order) pairwise matching. 
We formalize this idea as follows: 

Without loss of generality, assume the affinity-wise 
super graph Gsup fully connected which is induced by 
the configuration X. Given Gi, Gj, all loop-free matching 
pathes on the super graph can form a set of loop- 
free paths Zy = {Zij{ki ^..., ks)}^Si, whose cardinality 
\Zij\ = Thus the overhead for finding the 

highest score solution is intractable since s\ times 

of compositions need be computed. Given the matching 
configuration one alternative is approximating 

the optimal Xy by a series of iterations involving the 
first-order paths. And the problem of finding the optimal 
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third-party graph Qk in iteration t becomes: 

fc* = arg max J (4) 

This can be regarded as approximately concatenating 
the iteratively generated piecewise pathes, into a higher- 
order one that consists of multiple intermediate graphs. 

The above idea is concretized into an iterative al¬ 
gorithm termed as Iterative Score Boosting (ISB) as 
described in the chart of Algjl In iteration t, every 
is updated by seeking the path with order 5 = 1 via 
maximizing the affinity score according to Eq|^ The effi¬ 
cacy of such a score-boosting strategy can be justified by 
an intuitive analysis: even matching graph Qi and Qj is 
inherently ambiguous when both are largely corrupted, 
it is still possible to improve matching accuracy by an 
intermediate graph Qk that can find the quality pairwise 
matches between {Qi,Qk} and {Qk,Qj} respectively. The 
following proposition depicts the convergence of Alg[^ 

Proposition 1. Alg^ (ISB) is ensured to converge to a 
stationary configuration X* after a finite number of iterations. 

Proof: For each JiXij), it forms a non-descending 
sequence over iterations which is bounded by the upper 
bound Jij = max{vec(Y)^KijVec(Y),Y G in the 

discrete permutation space < • • • < 

‘ < Jij- Thus will converge. □ 

In fact, Algj^ (ISB) cannot guarantee the convergent 
X* satisfy full consistency i.e. (7(X*) = 1 by Definition 
Thus, after the iteration procedure, a post step for 
enforcing full consistency is optionally performed. In our 
heuristical implementation, when the resultant C(X*) 
is less than a given threshold 7, which suggests there 
are many contradictory matchings Xij 7 ^ thus 

the consistency is too low to be informative. Then the 
affinity-wise super graph by Definition (I^ is built 
to generate the final fully consistent X* in me hope 
that affinity score might be more indicative. Otherwise, 
we utilize consistency to smooth the final solution in 
two cases: i) when N > n i.e. the number of graphs is 
larger than the number of nodes, then the method in 
is adopted to obtain X*; ii) otherwise, the consistency- 
wise super graph Qc by Definition 0 is built to obtain 
X*. We apply this policy in all our algorithms for post¬ 
processing to achieve full-consistency. Yet how to realize 
full consistency in post-step is not the focus of this paper 
which has also been addressed by other work such as 
m m Moreover, in the presence of a considerable 
number of outliers, enforcing the full consistency will 
cause performance degeneration due to the unmatchable 
outliers as will be shown in our experiments. 

In order to obtain a comprehensive view of the affinity 
boosting model, we consider two additional variants of 
ISB (Alg[^. One uses the pairwise consistency instead 
of affinity to update solutions. The other replaces the 
first-order path selection with a second-order one. In 
fact, there can be other random search strategies aside 
from the fist-order one described here, and these random 


search variants can be derived by our framework. 

The first variant involves ri^lacing the updating step 

using EqjJwith the pairwise consis¬ 
tency in Definition (|^, as formulated as follows: 

k* = argmaxC'p(x7^’x7^\x^‘“^’) (5) 


Unlike ISB, this consistency-driven variant ISB^®^ (note 
superscript added) cannot ensure to converge to a 
stationary configuration. In particular, the consistency 
boosting mechanism at each round, can only guaran- 
tee C'p(xg\x(*-i)) > for any pair 

of i, j. Nevertheless, it cannot ensure Cp{xfj > 

Thus the non-decreasing property 
Cpix\f< Cp(x7,X(i)) < ... does not hold, 
although they are bounded by Cp{xfj < 1 . For 

similar reasons, one cannot ensure C{XJ^) > C{XJ~^^) 
where C(-) is defined in Definition As the solution 
space is discrete and finite, ISB^®^ either converges to a 
stationary point or forms a looping solution path. The 
former case is much more often observed in our tests. 

The 2 nd-order variant ISB^^^^ is derived by replacing 


^ij — ^ik ^kj ^ij ~ ^iv ^vu ^uj 


rP) _ 


u — arg max y K^^y, y = vec(X, 

u,v = l 


(t—1) y(^—1) y(^~^) 


( 6 ) 


The 2nd-order method has more exploration capability 
than the first-order one, at the expense of growing 
searching space from 0{N) to 0{N‘^) in terms of travers¬ 
ing the rest graphs (or graph pairs for the 2 nd-order 
case). As the same with ISB, the 2nd-order method can 
also guarantee convergence due to its score boosting 
property. These three methods are evaluated in Figj^ 
together with Alg|^ (ISB-GC) as will be introduced later. 
Moreover, in the ideal case when the ground truth is 
know, then we modify the method ISB by boosting the 
true pairwise accuracy instead of the affinity score for 
iterative updating. It is termed as ISB^^^ and shown in all 
the plots which can be regarded as the accuracy upper- 
bound given the initial X^o). 

One can see that ISB outperforms ISB^®^ notably. This 
is because ISB^®^ is based on the assumption that correct 
node-to-node matchings are dominant and the meaning¬ 
ful correspondences can be realized along many different 
paths of maps. This assumption would break given 
an unsatisfactory initial X( 0 ), which in turn calls for 
an affinity boosting step to satisfy the assumption. For 
ISB^nd, outperforms the first-order method, while its 
overhead becomes significantly larger when more graphs 
are considered. We leave the two methods Alg|^ (ISB- 
GC) and its variant ISB-GC^^^ which also both appear in 
Figto the next sub-section. 


3.3 Graduated consistency-regularized boosting 

Our key rationale is viewing the consistency constraint 
as a regularizer for affinity score maximization. Note that 
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Algorithm 1 Iterative Score boosting (ISB) 

lN-l,N 


Input: Tmax, ( 5 , 7 G ( 0 , 1 ); 


Perform pairwise matching to obtain initial 
Calculate vec(x7)^K,,vec(x7); 


max 

for alH = 1, 2,..., iV - 1 ; i i + 1,..., iV do 
update Xg^ = X^‘“^’x7^’ by solving EqQ 

end for 

end for 

if C'(Xb)) = 1, return X* = X^); 
if C(Xb)) < 7 then 

Build the super graph Qsup by pairwise affinity score 
and find a maximum span tree to generate X*; 

else 

if n > iV then 

Build the super graph by and find 

a maximum span tree to generate X*; 

else 

Perform the smoothing method 1^ to obtain X*; 

end if 
end if 


Algorithm 2 Graduated Consistency-regularized Iterative 
Score Boosting (ISB-GC) 


Input: 


s, 7; 


ij j i=l^j=i-\-l/ 

Perform pairwise matching to obtain initial 

Calculate vec(x7'^ 

for t = 1 : To do 

for alH = 1, 2,..., — 1; j = z + 1,..., iV d( 


Lvec(x7) K*jvec(y 


( 0 )') 
ij J 




kj 


by solving Eq 


update X,^ 
end for 
end for 

for t = To + 1 : Tmax do 

for alH = 1, 2,..., — 1; j = z + 1,..., iV d( 

update by solving Eq[ 

end for 


-x: 


(i) I 


< 6 , break; 


= ?Tzzn(l, 

end for 

Perform the same post-processing as in Algp](l |i0{18[ . 


maximizing pairvYise matching score among all pairs 
cannot ensure the consistency constraint. Moreover, due 
to outliers, sparse edge sampling for efficiency, and local 
deformation as well as the difficulty in parameterizing 
the affinity matrix as already discussed in [Jl and the 
references therein, there can be a case that for some 
pairs of graphs, the semantic ground truth matching 
configuration may not correspond to the highest affinity 
score. Thus purely maximizing the overall matching 
score is biased to accuracy, though maximizi^ overall 
consistency alone is even more biased as Algl^®* which 
has been studied in the previous sub-section. Informally 
speaking, this is analogous to loss function modeling 
in machine learning, where one not only considers em¬ 
pirical loss on the training dataset, but also introduces 
regularizer to account for over-fitting. 

One shall note as a baseline method, ISB separates 
affinity score maximization and consistency smoothing 


into two independent steps. It is yet appealing to tackle 
the two aspects jointly. We make the following state¬ 
ments, for devising an effective algorithm that gradually 
introduces consistency during the score-boosting proce¬ 
dure. It is motivated by two observations: i) For the 
initial assignment matrix X*^^^ obtained by the pairwise 
graph matching solver, its scores are more informative 
for the true accuracy; ii) After several rounds of it¬ 
erations for score-boosting, affinity score becomes less 
discriminative and consistency becomes more indicative. 

In this spirit, we infuse the matching consistency by a 
weighted term, whose weight A is gradually increased. 
As a result, the valuation function in each iteraton is 
changed from Eqfflto a weighted on^between Eqffland 

Eq||by setting 

r = argmax(l - + AC'p(Y^^T^\X^‘“^>) ( 7 ) 


The similar post-processing that enforces full consis¬ 
tency in Alg[^ (ISB) can also be conducted here which 
will stop the score growing immediately. This method is 
detailed in the chart of Alg|^ Graduated Consistency- 
regularized Iterative Score Boosting (ISB-GC). 

Similar to ISB^®^ ISB-GC in general cannot guarantee 
to converge to a stationary configuration, since the non¬ 
decreasing property of the sequence 
cannot always hold. Our empirical tests show this 
method can often converge to a stationary X* after 10 
iterations or so. In our experiments, we stop it when the 
number of iterations arrives a certain threshold T^ax- 
To make our study more comprehensive, like ISB^®^ 
to ISB, we further devise a counterpart to ISB-GC by 
swapping the role of consistency and affinity in this 
algorithm: the iterative updating is driven by choosing 
the anchor graph that lifts the pairwise consistency 

Cp(X-^\ as denoted by EqJ^for the method ISB^^k 

Meanwhile, the weight of the afmiity score is gradualty 
increased as the same as the consistency used in ISB-GC. 
Accordingly, the evaluation function becomes: 

k* = argmaxAJ(Y7^^) + (1 - (8) 

We call this algorithm ISB-GC^^^ where the superscript 
denotes for 'inverse'. The parameter settings A^ and P 
are identical to ISB-GC. We omit the algorithm chart for 
ISB-GC^^^ since it is akin to ISB-GC. It is outperformed 
by ISB-GC as shown in Eigj^ which validates our idea. 

3.4 Efficient consistency-regularized boosting 

The step of computing in ISB- 

GC need be repeated for each new k because it depends 
on the dynamically generated candidate 
a result, its time complexity regarding the consistency 


2. Note the affinity score is not directly comparable with either 
the unary or pairwise consistency as the latter fall in [0,1] while 
the former is arbitrary depending on how the affinity matrix is 
set. In our implementation, the affinity score is further normalized 
by J{Xij) = J(Xij)/max^jl J(X^°^)} where the denominator is a 
constant. This convention is also used when is introduced in Sec 3.5 
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- ISB* ISB — ISB*°^^ — ISB°^^ — ISB-GC* — ISB-GC - ' ISB-GC*"^""^ — ISB-GC'^''^—RRWM— 




(f) Deform by N 


(g) Outlier by N (h) Density by N (i) Coverage by N (j) Deform by N (time) 


Fig. 1: Comparison of ISB, ISB-GC and ISB-GC^’^'" on the synthetic random graph data by varying the noise (top row) and by the number of graphs 

(bottom row). RRWM Qi] is used as the pairwise matcher to generate the initial confi guratio n ('RRWM')- TSB’^^^^' denotes the ideal 'upper-bound' by boosting true 
accuracy rather than affinity or consistency. 'ISB*^' denotes ISB dismissing the step of l |l0|18j so for other methods. Refer to Table^for the settings. 


— ISB* — ISB ISB-GC-U* ISB-GC-U—-ISB-GC-P* ISB-GC-P — ISB-GC* — ISB-GC — RRWM—ISB*^ 


ISB* 







(a) Deform by T^ax (b) Outlier by T^ax (c) Density test by T^ax (d) Coverage by Tmax (e) Outlier by T^a,aL_(bi^e)(f) By search sample rate 
Fig. 2: Performance on synthetic data by varying the iteration threshold Tmax tor the proposed algorithms. Refer to Tablel^for the corresponding settings. 


TABLE 1: Settings for synthetic random graph test in Fig^and Fig[^ 


fixed parameters 


£=.08-.18 7V=30,ni=10,no=0,p=.9,cr=.05,c=l,Tmaa:=6 

77,0=1-6 A=30,£:=.05,7T,i=8,p=l,cr=.05,c=l,Tr„acc=6 Fig [i(b), Fig 3157 

P=.45-.6 A=30,7T,i=10,?T,o=0,£=.05,Cr=.05,C=l,Tmaa;=6 Fi| 

c=.05-.3 A=30,?7,i=10,?T,o=0,g=.05,p =l,(T=.Q5,Tmax=^ Fig| 




Fig 31^ 
Fig 


A=4-32 ni=10,no=0,£=.15,p=.9,o-=.05,c=l,T,nax=6 Fi gTO] Fi; 

A=4-32 7T,i=6,7T,o=4,£=0,p=l,cr=.05,c=l,Trnax=6 Fig iT^ Fig 

A=4-32 ?T,i=10,7T,o=0,£=0,p=.5,cr=.05,c=l,Tmax=6 Fig TTKI Fig 

A=4-32 ?T,i=10,no=0,£=.05,p=l,cr=.05,c=.l,Tr,iaa;=6 ~ ~ 


_ mr 

FipiflFigigiTT 


TABLE 2: Settings for synthetic random graph test in Figg] 


x-axis test mode fixed parameters rej 

mlts 

Tmax=0-6 deform N=20,ni=10,no=0,e=.15,p=.9,c=l,a=.05 Fig 

Tmax=0-6 outlier A=20,ni=6,no=4,£=0,p=l,c=l,cr=.05 Fig 

Tmax=0-6 density N=20,ni=10,no=0,e=0,p=.5,c=l,a=.05 Fig 

Tmax=0-6 coverage N=20,ni=10,no=0,e=.05,p=l,c=.l,a=.05 Fig 

sample rate=0-l mixture N=20,ni=8,no=2,e=.l,p=l,c=l,a=.05 Fig 

2(a) 

2mT 

in 

pr 


term is 0{N‘^n^) for updating each and the overall 
cost is 0{N^n^) in each iteration of ISB-GC. To reduce 
this overhead, we are motivated to use two more efficient 
consistency metrics to delegate 

The first proxy metric is the unary consistency as 
described in Definition Note this metric is also what 
we adopt in our preliminary work ||l] where it is referred 
as 'Algorithm 2'. Its evaluation function is: 

k* = argiMx(l - -|- XCu{k,X^*~^^) (9) 

The merit of this metric is all can be 


pre-computed in each iteration only for once. 

Alternatively, we devise another consistency estimator 
by first pre-computing all 

in the beginning of each iteration t — 1, and 
then adopting Eq[^ in the hope of approximating 

reused in updating 

fc* = argmL(l - A) JtX^L'(10) 

We term the above two efficient variants as 
Unary/Pairwise Graduated Consistency-regularized It¬ 
erative Score Boosting as depicted in the chart of Alg|^ 
In this paper, they are further abbreviated by ISB-GC- 
U and ISB-GC-P, where the last characters denote for 
'unary' and 'pairwise' respectively. In particular, ISB-GC- 
U has a convergence property as stated in below. 

Proposition 2. ISB-GC-U will converge to a stationary X*. 

Proof: Given two graphs Qi, Qj of n nodes for 
each, first, define the set of score difference as 

AJij = |vec(X)^Kiyvec(X) - vec(Y)^Kijvec(Y)|, VX, Y, 
between any two assignment matrices X, Y G in 

the enumerable permutation space. Suppose the largest 
value of difference is AJij which is constant given 
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Algorithm 3 Unary/Pairwise Graduated Consistency- 
regularized Iterative Score Boosting (ISB-GC-U/P) 


Perform pairwise matching to obtain initial 


Run I 3 l 7 


jvec{X\ 


for t — iQ “h 1 I 1 'ma 


Input: 

1 : 

2 

3 

4 

5 

6 

7 

8 
9 

10 
11 
12 

13 

14 

15 

16 

17 

18 

19 

20 


r, (5, 7 , s, mode; 


Calculate vec(x7)- 

in Alg|2| for pure affinity score driven boosting; 

do 


(0)^ 


max 

then 


if mode='unary' 

Compute unary consistency {Cu{k,^. 
else if mode='pairwise' then 
Update pairwise metric 

end if 

for alH = 1, 2,..., — 1; j = z + 1,..., iV do 

if mode='unary' then 

update by solving 

else if mode='pairwise' then 


)U=l,j=i+l' 


update Xf^ = 

end if 


end for 
if 

= mini 


by solving Eq 


< 5, break; 
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-1 W^ij 

end for _ 

Perform the same post-processing as in Algp](l |l0{18[ . 


Kijj We further define the largest difference denoted by 
AJg = inax{AJi77i,iA+i for all pairs in the graph. 

For a certain iteration in ISB-GC-U, suppose the most 
consistent graph by Definition (IB is Qa/ and the second 
one is Gb- In iteration t it will finffly satisfy the condition: 
ACi*'’ = C„(a,Xb)) - C„{6,Xb)) > ^-^AJg as ^ 1 
by Ab) = iH Then, the following inequality 

will hold for any k and 

(1 - A)J(Xb>Xib) + AC4a,xb-b) (11) 

=(1 - A)J(Xb>xW) + AC'„(6,xb-b) + aaC^ 

=(1 - A) (j(Xb)xib) - J(ybxW)) + (1 - A)J(xWxb>) 

+ AC'„(6,xb-b) + AAC'7 

^AAC^ - AJg + (1 - A)J(ybxb;) + AC'„(fc,xb-b) 

>(1 - A)J(Xb)xi7 + AC4fc,xb-b) 

As a result, all will be updated by 

in iteration t, Xb) become fully 
consistent by Definition (j^ thus cannot be lifted by either 
affinity or consistency metrics used in this paper. □ 
The iterative procedure in ISB-GC-U may exceed the 
maximum iterating threshold T^ax before reaching the 
above inequality holds when A is assumed close to 1 
enough. Thus a post-processing step is also needed in 
line with Alg[^ (ISB) and Alg|^ (ISB-GC) if necessary. 

For the 'pairwise' version ISB-GC-P, this method in 
our experiments also often converges to a stationary X* 
similar to ISB-GC, yet there is no theoretical guarantee. 
Fig|^ illustrates how the overall matching accuracy is 
lifted via the proposed algorithms, controlled by Table 

3. In case Cu(a,X(^)) = 67^(6, without loss of generality, one 

can choose any of them as the largest one, and choose the next Qc as 
the second largest if Cu{a,Xi*^) > C'n(c, X^^)). 


3.5 Eliciting affinity and consistency for inliers 

One limitation of many previous studies for both pair¬ 
wise and multi-graph matching as mentioned in this 
paper, is that they enforce all nodes in one graph to 
have matchings from the other(s), either by explicitly 
using a permutation matrix imposing strict node-to-node 
correspondences (471 , ||13|, or implicitly doing so by 
generating node-to-node matchings as many as possible 
such that the objective affinity score is optimized 03, 
lEI, EH, ED, Eol. A common simplification when 
applying a pairwise GM method is assuming one of two 
graphs is a reference graph, such that each of its node 
can find a match from the other testing graph. 

We are motivated to devise a general outlier-rejection 
mechanism not depending on the particular characters of 
the testing data in terms of the node distribution, weight 
attributes, noises type, and graph structure etc. 

The Graduated Assignment method [50l attempts to 
handle outlier nodes by assigning them to additional 
slack variables. While in the presence of a majority of 
outliers, this approach is not widely adopted due to 
its sensitivity to parameters for the slack variables. The 
very recent work [3I| proposes a general approach via 
a max-pooling mechanism suited for the scenario of 
matching two graphs with a majority of outliers. A less 
relevant work is by Torresani et al. 143 which also allows 
nodes to be assigned in an unmatchable status in an 
energy function induced on the Markov Random Field. 
However, its complex objective function is designed that 
account for various similarity measurements e.g. appear¬ 
ance descriptors, occlusions, spatial proximity etc. be¬ 
yond the general setting of the affinity matrix considered 
in this paper. Moreover, the associated parameters need 
to be learned with labeled correspondence ground truth 
from a training dataset which impedes its applicability. 
Other pairwise GM methods 1431 , |i42l integrate the point 
detection and matching in a synergic manner by an 
integrated system, while this paper assumes the graphs 
are given and specifically confined in visual problems. 

As a building-block, all these pairwise GM methods 
can be used by our approaches and other state-of-the- 
arts 12, l29l , Il4|, l28l to improve the initial X^^^. Never¬ 
theless, the context of multiple graphs allows for a more 
robust mechanism for pruning outliers such that only 
the affinity and consistency relevant to common inliers 
are considered in our boosting framework. 

We describe our 'inlier nodes' eliciting method us¬ 
ing the node-wise consistency Cn(i^^,X) by Definition 
(|^. Another alternative is using the node-wise affinity 
Sn{u ^via Definition (|^, which can be applied in 
a similar fashion thus its description is omitted here. 

First, given the matching configuration X, each node 
in one graph is scored by the node-wise consistency by 
Definition Assume the number of common inliers 
i.e. rii for all graphs is known, which is available when a 
reference template is given, or estimated by other means 
e.g. m How to estimate rii is not the focus of this paper. 
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TABLE 3: Description of the parameters for experiment settings in this paper. 


graph# 

inlier# 

outlier# 

estimated Ui 

deform 

density 

sensitivity 

coverage 

iter# 

N 

Ui 

no 

nest 

£ 

P 

a 

c 

Tmax 


Then we make two revisions for all of our methods: 

i) the pairwise affinity term vec(X)^Kvec(X) is modified 

by keeping the rows of X that correspond to the first 
rii (the number of inliers) nodes in descending order by 
their node-wise consistency score Cn , X) in Definition 
ffl, and zeroing the rest of rows. This is because the 
fflinity or consistency between outliers is irrelevant to 
the semantic matching accuracy thus shall be excluded 
in the boosting procedure. For node-wise consistency, we 
use V^c(X, X, Ui) (or V^a(X, X, K, rii) for node-wise affinity) 
to denote this 'mask' operation on X as it is determined 
by both the input configuration X and rii (also K in 
case of the affinity metric). Then the affinity score is 
rewritten as = vec('0c(^5 ^i))^I^vec('0c(^5 ^i)); 

ii) the consistency term is modified by the following two 
equations for unary and pairwise consistency: 


( 12 ) 

Ef=TEf=.+i 


=1 - 


niN{N - 1 ) 


(13) 


= 1 - 


2niN 


We use the above tailored variants for affinity score 
and consistency to replace the original J, Cu and Cp in 
the evaluation functions: Eq|^ Eq|^ Eq|^ Eq[^ Similar 
steps are performed for using the node-wise affinity. 


4 Experiments and Discussion 

The experiments are performed on both synthetic and 
real-image data. The synthetic test is controlled by vary¬ 
ing the noise level of deformation, outlier, edge den¬ 
sity and initial pairwise matching coverage. The real 
images are tested by varying viewing angles, scales, 
shapes etc.. The matching accuracy over all graphs, is 
calculated by averaging all pairwise matching accuracy 

y^N—l y^N , , 

Each AcCij computes the matches be¬ 
tween given by the ground truth X^^^: Acc^j = 

tr(i X [4J, |T3I, we only calculate the 

accuracy for common inliers and ignore the correspon¬ 
dences between outliers. The above protocol is widely 
adopted by related work such as GH, |[l3l . 

If not otherwise specified, the parameters of our meth¬ 
ods are universally set by: To = 2, Tmax = 6, = 0.2, 

7 = 0.3, (3 = 1.1. Note when t < To, only score boosting is 
performed without infusing consistency regularization. 
As a simple speed-up trick used in our methods, for the 
generated candidate solutions in the evaluation formulas 
Xij by Eq|^ and Eq|^ we compute their scores after 
excluding those duplicate ones. 


4.1 Dataset description and affinity setting 

Synthetic random graph matching The random graph 
test follows the widely used protocol of 12^ , ||20|, ft^ . 


ini, 1^ . Eor each trial, a reference graph with rii nodes 
is created by assigning a random weight to its edge, 
uniformly sampled from the interval [0,1]. Then the 
'perturbed' graphs are created by adding a Gaussian 
deformation disturbance to the edge weight which 
is sampled from N{0,£) i.e. = g[^+A(0,5) where 
the superscript 'p' and 'r' denotes for 'perturb' and 
'reference' respectively. Each 'perturbed' graph is further 
added by rio outliers, which can also be helpful to make 
the graphs of equal sizes when the input graphs are 
different sizes. Its edge density is controlled by the 
density parameter p e [0,1] via random sampling. The 
edge affinity is computed by Kiajb = exp(—) 
where is the edge similarity sensitivity parameter. No 
single-node feature is used and the unary affinity Kia^ia 
is set to zero, leaving the matching score entirely to the 
pairwise geometric information. In addition, we control 
the 'coverage' of the initial matching configuration X by 
parameter c G [0,1], which denotes the rate of matching 
pairs that are generated by the pairwise matching solver, 
and the rest 1 — c portion of pairs are assigned with a 
randomly generated matching solution. A description of 
these parameters is listed in Table 

Synthetic random point set matching The random 
point set matching is also explored as tested in If3l , 
Il39l . Eirst, rii inliers are randomly generated 

on the 2-D plane via Gaussian distribution A^(0,1) as 
the reference point set. Then each point is copied with 
Gaussian noise A(0,5) to generate N random point 
set by further adding rio outliers via A(0,1). The edge 
weight is set by the Euclidean distances qij = ||p- — Pj|| 
for each graph. The subsequent steps are the same with 
the random graph matching case. This dataset is used for 
the synthetic testing in comparison with MPM |3| under 
a relatively large number of outliers. This is because the 
max-pooling method MPM is designed for geometric 
relation rather than arbitrary edge weights as in the 
random graph matching setting. 

CMU-POSE Sequence This data contains four se¬ 
quences. Two sequences are from the CMU house (30 
landmarks, 101 frames), hotel (30 landmarks. 111 frames) 
sequence (http://vasc.ri.cmu.edu//idb/html/motion/) 
which are commonly used in ||29l, GH, GD, G3l , 
1321 . The other two sequences are sampled from the 
'VolvoCZO' and the 'houseblack' (both 19 landmarks, 225 
frames) covering a range of 70 degrees of viewing angles 
from the POSE dataset l4^ . We use this data for the 
'partial similarity' test. We select n^^lO landmarks out 
of all riant annotated landmarks, and randomly choose 
77-0=4 nodes from the rest riant —nodes as 'outliers'. 
The edge sampling follows the same way as G^ by 
constructing the sparse delaunay triangulation among 
the points (no distinction to inliers or outliers). The 
affinity matrix is set by Kiajb = where 

dij, dab are the Euclidean distances between two points 
normalized to [0,1] by dividing the largest edge length. 
The diagonal is set to zero same as GH/ G3l . 
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TABLE 4: Time complexity comparison of the state-of-the-arts. 


algorithm 

time complexity 

ISB (Algl^, ISB-GC-U, ISB-GC-P (Alg^ 

ISB-GC (Algm 
MatchLift 0 
MatchOpt (non-factorized) 1291, fil 
MatchSync 1281 

0{N^n^ + ATVpai,) 
0(N‘'n + NV + AT^Tpair) 
0{NV + ATVpair) 
0(N^n‘^ + N^n + AT^Tpair) 
0{N^n^ + ATVpair) 


WILLOW-ObjectClass The object class dataset re¬ 
leased in 1321 is constructed by images from Caltech-256 
and PASCAL VOC2007. Each object category contains 
different number of images: 109 Face, 50 Duck, 66 Wine 
bottle, 40 Motorbike, and 40 Car images. For each image, 
10 feature points are manually labeled on the target 
object. We further add Uq random outliers detected by 
a SIFT detector in our outlier test. The edge sampling 
for the affinity matrix follows the same way as 11311 by 
constructing the sparse delaunay triangulation among 
the landmarks as done in the CMU-POSE sequence test. 
For defining the edge affinity, we follow the protocol of 
lHa, EH that sets the final affinity matrix re-weighted 
by the edge length affinity and angle affinity: Kia^jb = 
PK^ia,jb + (1 “ where (3 e [0,1] is the weighting 

parameter. This is because this dataset contains more 
ambiguities in terms of structural symmetry as pointed 
out in [4|, than the sequence dataset if only length infor¬ 
mation is used. The angle for each edge is computed by 
the absolute angle between the edge and the horizontal 
line as used in |T3|. The edge affinity and angle affinity 
are calculated in the same way in the CMU-POSE test. 
For node-wise affinity is also set to zero to avoid the 
point-wise feature dominate the overall matching results. 

For modeling affinity, this paper considers the second- 
order information, yet higher-order edge affinities in the 
hyper-matching setting 1361 , ll3TI are also applicable in 
our methods since their affinity score can be computed. 

4.2 Comparing methods and time compiexity 

The implementations of all comparing methods are the 
authors' Matlab code and all tests run on a laptop (2.9G 
Intel Core 17 and 8G memory) with a single thread. 

Re-weighted Random Walk Matching (RRWM) Since 
many existing multi-graph matching methods Il29l , 

EH/ [21/ Il26l . >411 require a pairwise GM solver in 
different ways, we choose RRWM ||14| due to its cost- 
effectiveness. We set its parameters n=0.2, ^5=30. 

Max-Pooling Matching (MPM) MPM [3| is an outlier- 
tolerant method computing the affinity score of each can¬ 
didate match via maximal support from nearby matches. 
This method is tested in the presence of more outliers. 

Several state-of-the-art multi-graph matching methods 
are evaluated together with our method^ It shall be 

4. Due to space limitation, we have to omit a few peer multi- 
graph/point-set matchi ng m ethods, including graduated assignment 
based common labelir^ |45l, (SO), the joint feature point matching |50l 
and the recent work (9J. The method in |45l, 1^ has been evaluated in 
( 31 , and | 2 l is a more advanced method originated from 19 ). The feature 
matching method |50l only considers point-wise features descriptors 
rather than the 2nd/higher-order information as formulated in the 
GM problem. Thus in fact this becomes a first-order point matching 
problem rather than graph matching. 


noted that all methods start with an initial matching 
configuration obtained from RRWM in this paper. 

Alternating optimization for multi-graph matching 
(MatchOpt) MatchOpt ||29l, |4j transforms the multi¬ 
graph matching problem into a pairwise one by a star 
structure, whose root is the reference graph Qr, and 
the leaf are the others {Qk}k=i,j^r- Then it sequentially 
updates each Xm by fixing the rest N — 2 basis variables. 
Both factorized and non-factorized models regarding the 
affinity matrix are devised in i4j. Only the non-factorized 
model MatchOpt is compared as it has been shown 
in m more cost-effective. We set its iteration threshold 
Tmax = 4 as it often converges quickly. 

Match lifting via convex relaxation (MatchLift) 
MatchLift ||2| adopts a first-order approximate algorithm 
for semi-definite programming. The main cost is com¬ 
puting the eigenvalues for iterative updating the dual 
variable at cost of 0{N^n^). This solver typically requires 
more iterations to reach optimum results (10-20 rounds) 
thus we set Tmax = 20. 

Permutation synchronization (MatchSync) Match- 
Sync ||28l uses spectral analysis to the grouped 'target 
matrix' T of size nN x nN (Eq.7 in 1 ^1 ) stacked by all 
initial It involves a one-shot SVD to find the 

n leading eigenvectors of'T' whose cost is 0{n^N‘^), and 
N iterations of Hungarian method with cost 0{Nn^). 
Thus the total cost except the initial pairwise matching 
step is 0{N‘^n^). Since MatchSync is applicable only 
when n < N, thus the Maximum Spanning Tree (MST) is 
used on the consistency-wise super graph iin> N. 

We consider the time complexity of our methods in 
several aspects as follows. The overall time complexity 
for all compared methods is summarized in Table 

First, computing the pairwise score JiXikXkj) in all 
our methods by trying N — 2 anchor graph ^ 

is repeated for pairs, at the unit cost of O(n^) as 

Xij is a sparse permutation matrix. Thus the related over¬ 
head is 0{N^n^), which is also the complexity of ISB, 
except for computing the initial X^^^, whose complexity 
O(A^^rpair) depends on the pairwise matching solver. 

The second category refers to the computing of the 
pairwise consistency {Cp(X^/cX/ej, X)}^^ in Algorithm 
ISB-GC. The complexity of Cp is 0{Nn) by code level 
optimization, therefore similar to the first category, the 
total overhead in iteration t is 0{N^n), and the overall 
complexity of ISB-GC is 0{N^n)TO{N^n^)TO{N‘^Tp^ir)- 

The third category refers to the off-line computing 
unary and pairwise consistency as used in ISB-GC- 
U/R The cost for computing each unary consistency 
Cu can be reduced to 0{N‘^n), thus the overall cost 
of computing {Cn(/c,X)}^^ for pairs of graphs 

is 0{N^n). As a result, the overall time complexity of 
ISB-GC-U is 0{N^n^) + 0{N^n) + O(A^Vpair). On the 
other hand, computing the off-line pairwise consistency 

will cost 0{N^n) at the unit 
expense of 0{Nn) for each Cp. The overall complexity 
of ISB-GC-P is also 0{N^n^) + 0{N^n) + O(A^Vpair). 






JUNCHI YAN ET AL A GENERAL MULTI-GRAPH MATCHING APPROACH VIA GRADUATED CONSISTENCY-REGULARIZED BOOSTING 


11 


— ISB* — ISB ISB-GC-U* ISB-GC-U ISB-GC-P* ISB-GC-P — ISB-GC* — ISB-GC —RRWM — ISB*^°°^MatchSync — MatchOpt ^MatchLif t 



(a) Deform by e 


outlier# 

(b) Outlier by n, 


0.15 0.2 

complete 


(c) Density by p 


(d) Coverage by c 


outlier# 

(e) Outlier by Uq (time) 



12 16 20 24 28 32 

graph# 


12 16 20 24 28 32 

graph# 



(f) Deform by N (g) Outlier by N (h) Density by N (i) Coverage by N (j) Deform by N (time) 

Fig. 3: Evaluation for ISB, ISB-GC and the two derivatives ISB-GC-U and ISB-GC-P, and state-of-the-arts on the S 5 mthetic random graph dataset by varying the 
disturbance level (top row), and by varying the number of considered graphs (bottom row). Refer to Table^for the settings. 

— ISB* — ISB ISB-GC-U* ISB-GC-U-—ISB-GC-P* ISB-GC-P — ISB-GC* — ISB-GC —RRWM — ISB*^°°^MatchSync ^MatchOpt ^MatchLif t 




(c) Motorbike by N 


(d) Winebottle by N 


graph# 

(e) Winebottle by N (time) 




graph# 


graph# 


graph# 


(f) Hotel by N (g) House by N (h) VolvoCZO by N (i) Houseblack by N (j) Houseblack by N (time) 

Fig. 4: Evaluation for ISB, ISB-GC and the two derivatives ISB-GC-U and ISB-GC-P, and state-of-the-arts on the real image data. Refer to Table|^for the settings. 


TABLE 5: Parameter settings for real-image test in Fig[4| 


object x-axis fixed 

rparameters 

car, duck, bike, bottle iV=4-28 c = 1, cr^ = .1, /3 = .9, n* 

= 

10, no = 2 

hotel, house, volvoCZO, houseblack ^=4-36 c = 1, cr^ = .05, /3 = 0, n* 

= 

10, no = 4 


Finally, we consider the run-time overhead 
for the inlier-eliciting mask operation 7/;c(', X, 
or '0a(', X, K, n^). For '0c(',X, n^), it is in fact 
concerning with computing the node-wise consistency 
{C'„(w'=,X)}”Ai for each node in every graph, 
thus the total overhead in each iteration is 0{N^n) at 
the unit cost of 0{N‘^n) per graph. For 7/;a(', X, K, n^), 
the node-wise affinity by Definition § is 0{N‘^n^). 
Regardless or ^pa is used, the associated overhead 
can be absorbed in the overall complexity as shown 
in Table 1^ We will evaluate these two inlier-eliciting 
metrics in the experiments. 


4.3 Experimental results and discussion 


For the 'RRWM' and 'MPM' curves in the plots, they 
are generated by applying the two methods on each 
pair of graphs independently. TSB*' denotes performing 
ISB without running the post step i jTQp^ in the chart 
of Algj^ and so for other methods. Fifty random trials 
are performed for all synthetic tests that generate the 
average results as plot in the figures. For real image data, 
20 random trials are sampled from the image set for each 
setting. For clarity, standard deviations are not plot and 
they are found relatively homogenous over curves. 


In case of few outliers Fig[^ gives a comparison 
with controlled noise level, for several putative methods 
in this paper besid es the main algorithms: {ISB, ISB^®* 


ISB^^^} in Sec 3.2 and {ISB-GC, ISB-GC^^^} in Sec 


3.3 
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(a) Outlier by no (b) Outlier by N (c) Deform by e (d) Outlier by nest (e) Outlier by no (time) 

Fig. 5: 'Outlier' test for our methods driven by the consistency (solid) and affinity (dashed) inlier eliciting mechanism on random points, controlled by Tablej^ 

— 138-60-0*®““ — ISB-GC-P*®““—-138-60*®“" — 138*°®^ I3B-60-U*°®^ — I3B-60-P*°®^ — 138-60*°®^ RRWM — ISB*^°° MPM — match3ync — matchOpt 





(a) Car by no 


(b) Duck by no 


(c) Face by no 


(d) Motorbike by no (e) Winebottle by no 




(h) Motorbike by N 





(k) Motorbike by nest (1) Winebottle by nest (m) Motorbike by nest (n) Winebottle by nest (o) Inlier hittins rate by no 

Fig. 6: 'Outlier' test for our methods driven by the consistency (solid) and affinity (dashed) inlier eliciting mechanism on images. Refer to Table^for the settings. 


TABLE 6: Settings for random point set test with more outliers in Figl^ 


x-axis 

fixed parameters 

resuTfs 

no=6-16 

iV=4-32 

£=0-.l 

nest=l-18 

N=20,e=.Q2,ni=6,p=l,(7=.Q5,c=l,Tmax=^ 
£=.Q5,ni=6,no = V2,p=l,(T=.Q5,C=l,Trnax=^ 
N=20,ni=6,no=12,p=.9 ,a=.05,c=l,Tmax=^ 
N=20,ni=6,no=l2,£=.05,p=l,a=.05,c=l,Tmax=6 

Fig 

Fig 

Fie 

Fig 

5(a) 

Mb) 

McT 

bW 



TABLE 7: Settings of Willow-ObjectClass test with more outliers in FigJ^ 


object 

x-axis 

fixed paramef^s 

car, duck, face, bike, bottle 

no=0-20 

c = 1, = .1, ^ = .9, nest = ni = 10,N = 28 

car, bike, bottle 

Ar=4-28 

c = 1, = .1, ^ = .9, neat = ni = 10, rio = 10 

car, duck, bike, bottle 

ne.t=2-20 

c = l,a^ = .l,p = .9, m = 10, no = 10,N = 28 


Fig|^ shows the accuracy boosting behavior as a function 
of Tmax and based on this plot, we set Tmax = 6. There 
is little noise regarding with the affinity function for 
the coverage test in Fig 2(d) thus pure affinity-driven 
approaches outperform. In other cases, ISB and 
under-perform and the latter even slightly degenerates 
as Tmax grows. Fig |2(f) I suggests the relation between the 


accuracy and the sampling rate r for the {N—2)^r graphs 
used from all graphs over the exhaustive searching 
space, to find the best anchor graph. Since our frame¬ 
work is applicable to other random search strategies, 
which might also depend on the specific applications, 
thus here we do not dwell on how to design an efficient 
random search technique. Our comments are as follows: 

i) The proposed main method ISB-GC outperforms 
the baseline ISB and state-of-the-arts in most cases for 
both synthetic random graphs (Fig|^ and real images 
(Fig|^, except for the 'coverage' test (Fig |3(d)[ Fig |3(i)L 
in which many pairwise matchings in the initial 
are randomly generated and a consistent X* is biased 
to the true accuracy. Specifically, the methods utilizing 
affinity information including and ours, outperform 
MatchLift m and MatchSync (281 that only use the 
initial matching configuration X(o) as input, when X(o) 
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is largely corrupted as controlled by coverage rate c. 

ii) As shown in the bottom row of Fig|^ in addi¬ 
tion with Fig|^ the accuracy in general increases as 
the number of graphs N grows, though there is some 
fluctuations especially for the test on Aar' as shown in 
Fig|4(a)| We think this is due to the inherent matching 
difficulty of the sampled images, which is evidenced by 
the baseline 'RRWM' whose performance also fluctuates. 
We find this phenomenon is more pronounced as shown 
in the bottom row of Fig|^ due to the larger viewing 
range when more frames are sampled. In fact the relative 
accuracy improvement against the RRWM baseline is 
increasing as N grows. This is in line with the intuition 
that more graphs can help dismiss local ambiguity. 

iii) For the 'partial similarity' CMU-POSE data test 
which is often the case in reality, as shown in the bottom 
row of Fig|^ without enforcing full consistency (dashed 
curve) is more effective than using this constraint (solid 
curve). It suggests the efficacy of our graduated con¬ 
sistency regularized approach against the two-step hard 
synchronization methodology from another perspective. 

In case of more outliers Our methods are tailored 
to the presence of a majority of outliers as described in 
Sec 3.5 The performance is depicted in Fig|^ and Fig|^ 
for the tests on synthetic point sets (inline with the set¬ 
ting for the compared method MPM [3|) and images, by 
varyii^ the number of outliers, graphs and the estimated 
inlier^ No post-synchronization is performed in the 
outlier tests. Discussions are presented as follows: 

i) In Fig 6 (o)[ we show an example of the hitting 
rate of the top rii nodes in descending order by the 
node-wise consistency (solid curve) and affinity (dashed 
curve) metrics against true inliers. Given such reasonable 
hitting rates (> 0.8), the suite of our inlier eliciting 
methods, i.e. the consistency-driven variant (marked by 
the superscript 'cst') and the affinity-driven one (marked 
by 'sim') in general outperform state-of-the-arts notably. 
Note in Fig |6(o)} the hitting rate for 'face' by the affinity- 
driven metric is robust against the number of outliers. 
This is because the landmarks on face form a very 
distinctive structure thus its affinity is more informative. 

ii) The sensitive test w.r.t. the disturbance of Ui is 
illustrated in Fig 5(d) and the bottom row of Fig|^ for 
synthetic data and real images respectively. The elic¬ 
iting mechanism boosts the accuracy compared with 
our original algorithms which in fact set rii equal to 
n = 20 with no discrimination to outliers. Moreover, the 
accuracy decline is relatively smooth around the exact 

= 10 in these plots which suggests the robustness of 
our methods given a rough estimation of 

iii) Two run-time overhead examples are plot in 
Fig 5(e) for the synthetic test and in Fig |6(j)| for the real 
image test. ISB-GC^ is more costive as we find it con- 


5. MatchLift is not plot in Fig|^ Fig|^as they are already very busy 
It is not tailored for the presence of a majority of outliers thus under¬ 
perform as show n in t he supp lemental material. And its cost is heavier 
as shown in Fig |3(e)| Fig |3(j)| as it takes more rounds of iterations to 
reach a satisfactory solution due to the first-order iterative procedure. 


verges slower and need more overhead in each iteration 
as discussed in the complexity analysis. Also, all the 
proposed algorithms can be parallelized more easily than 
other iterative methods like matchOpt, matchSync, be¬ 
cause updating each can be done independently. 

iv) MPM outperforms the baseline RRWM but does 
not perform as robustly as our methods. We think this 
is because MPM is suited under moderate noises in 
addition with the outliers. This is often not true in 
realistic scenarios and our settings - Table|^and Table 

v) In the synthetic point data test for Fig |5(a)| and 
Fig 5(b) the affinity-driven inlier eliciting variant outper¬ 
forms the consistency-driven one given the deformation 
5 < .05. However, as shown in Fig |5(c)| when 5 grows 
by fixing the number of outliers, the consistency-driven 
mechanism outperforms when 5 > .05. This is consistent 
with the motivation of this paper: consistency becomes 
helpful in the existence of large noises given few outliers. 


5 Conclusion 

In the paper, we propose general algorithms towards 
multi-graph matching by incorporating both affinity 
score and matching consistency in an iterative approxi¬ 
mate boosting procedure. The outlier-tolerance variants 
are designed by eliciting the affinity and consistency 
associated with inliers. The experimental results suggest 
that i) the method Alg|^ (ISB-GC) in general achieves 
more competitive accuracy compared with state-of-the- 
arts; ii) the methods Alg|^ (ISB-GC-U/P) in general 
improve the cost-effectiveness of Algj^ (ISB) especially 
on the real image data under arbitrary noises. 
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